Papers with prompting techniques
Salient Information Prompting to Steer Content in Prompt-based Abstractive Summarization (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to generate concise summarizations require extensive modifications to the architecture. |
| Approach: | They propose a lightweight model that can be finetuned to extract salient keyphrases from the source document to enhance ROUGE F1 and recall. |
| Outcome: | The proposed model can be finetuned to extract salient keyphrases without any LLM customization. |
NYT-Connections: A Deceptively Simple Text Classification Task that Stumps System-1 Thinkers (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models have shown impressive performance on various benchmarks, yet their ability to engage in deliberate reasoning remains questionable. |
| Approach: | They propose to penalize quick, intuitive "System 1" thinking by combining linguistic isolation with resistance to intuitive shortcuts to assess model's reasoning abilities. |
| Outcome: | The proposed model penalizes quick, intuitive “System 1” thinking, isolating fundamental reasoning skills. |
Unlocking the Planning Capabilities of Large Language Models with Maximum Diversity Fine-tuning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that LLMs struggle to generate valid plans in the automated planning domain due to weak System 2 competencies. |
| Approach: | They propose a method which uses a random sampling method to select diverse and representative data to enhance sample efficiency and the model’s generalization capability. |
| Outcome: | The proposed method outperforms baseline methods across scales and multiple benchmark domains. |
Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing prompting techniques for Large Multi-Modal Models (LMMs) focus on improving textual reasoning or leveraging tools for image preprocessing, lacking a simple and general visual prompting scheme to promote vision-language coordination. |
| Approach: | They propose a prompting scheme that scaffolds coordinates to promote vision-language coordination in Large Multi-Modal Models (LMMs) they overlay a dot matrix within the image as visual information anchors and leverage multi-dimensional coordinates as textual positional references. |
| Outcome: | Experiments on a wide range of vision-language tasks show the superiority of SCAFFOLD prompting over the textual Chain-of-Thought prompting. |
Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategies (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized the field of natural language processing . however, it has been shown that they lack systematic generalization, which allows to extrapolate the learned statistical regularities outside the training distribution. |
| Approach: | They propose to benchmark a LLM with two parameters to find out its performance . they compare it to a variant of the Transformer-Encoder architecture to find the same problem . |
| Outcome: | The proposed model outperforms the previous model on three algorithmic tasks with two parameters. |
What do Large Language Models Need for Machine Translation Evaluation? (2024.emnlp-main)
Copied to clipboard
Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orasan, Tharindu Ranasinghe, Fred Blain
| Challenge: | Existing research shows that large language models can perform better in machine translation tasks. |
| Approach: | They propose to use large language models for machine translation evaluations . authors explore what translation information is needed for LLMs to evaluate MT quality . |
| Outcome: | The proposed model performs comparable to fine-tuned multilingual pre-trained models. |
TrojanSQL: SQL Injection against Natural Language Interface to Database (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on text-to-SQL systems have not investigated its security aspects . however, how to implement such attacks remains an open question. |
| Approach: | They propose a backdoor-based SQL injection framework for text-to-SQL systems that uses boolean-based injection and union-based injecting techniques to exploit SQL injection vulnerabilities. |
| Outcome: | The proposed framework can produce harmful SQL statements invalidating user queries or compromise sensitive information about the database. |
Are Emergent Abilities in Large Language Models just In-Context Learning? (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models have been claimed to acquire certain capabilities without having been specifically trained on them. |
| Approach: | They propose a theory that explains emergent abilities by taking into account their potential confounding factors and rigorously substantiate this theory through over 1000 experiments. |
| Outcome: | The proposed theory proves that emergent abilities are not truly emergental, but result from a combination of in-context learning, model memory, and linguistic knowledge. |
Towards Efficient and Robust VQA-NLE Data Generation with Large Vision-Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for creating a vision question-answering with natural language explanations rely on human annotations that are time-consuming and costly. |
| Approach: | They propose a method that generates high-quality natural language explanations using LVLMs by using visual prompts. |
| Outcome: | The proposed method generates high-quality synthetic VQA-NLE datasets 20x faster than human annotations with minimal decrease in qualitative metrics. |
MacGyver: Are Large Language Models Creative Problem Solvers? (2024.naacl-long)
Copied to clipboard
Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas Griffiths, Faeze Brahman
| Challenge: | a new study examines the creative problem-solving capabilities of modern LLMs . it provides insight into the constrained problem- solving capabilities of both humans and AI . |
| Approach: | They use an automatically generated dataset to compare and contrast LLMs and humans to find out their creative problem-solving abilities. |
| Outcome: | The proposed dataset compares LLMs and humans in a constrained setting . it shows that humans excel in tasks they are familiar with but struggle with domain-specific knowledge . |
LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probing (2025.acl-long)
Copied to clipboard
Dario Di Palma, Alessandro De Bellis, Giovanni Servedio, Vito Walter Anelli, Fedelucio Narducci, Tommaso Di Noia
| Challenge: | Large Language Models (LLMs) have become central to NLP, demonstrating their ability to adapt to various tasks through prompting techniques. |
| Approach: | They probe the hidden layers of Large Language Models to identify where sentiment features are most represented and to assess how this affects sentiment analysis. |
| Outcome: | The proposed approach enables sentiment tasks to be performed with memory requirements reduced by an average of 57%. |
Towards Human Understanding of Paraphrase Types in Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Current paraphrase evaluations of language models use binary approaches, offering limited interpretability of specific text changes. |
| Approach: | They introduce a dataset of 800 sentence-level and word-level annotations by 15 annotators and a human preference ranking of paraphrases with different types. |
| Outcome: | The proposed model can generate simple APTs, but struggle with complex structures (e.g., subordination changes). |
Think Twice: Perspective-Taking Improves Large Language Models’ Theory-of-Mind Capabilities (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances to LLMs’ reasoning capabilities from simple yet effective prompting techniques such as Chain-of-Thought have seen limited applicability to ToM. |
| Approach: | They propose a two-stage prompting framework inspired by Simulation Theory's notion of perspective-taking to elicit Theory-of-Mind capabilities in Large Language Models. |
| Outcome: | The proposed framework shows that it is much more effective than existing prompts. |
POSQA: Probe the World Models of LLMs with Size Comparisons (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Embodied language comprehension emphasizes that language understanding is not only mental processing in the brain but also involves interactions with the physical and social environment. |
| Approach: | They propose to use a physical object size question to examine the extremity of large language models to test their embodied comprehension. |
| Outcome: | The proposed dataset shows that even the largest LLMs perform poorly under the zero-shot setting. |
“Well, Keep Thinking”: Enhancing LLM Reasoning with Adaptive Injection Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit strong reasoning abilities, often attributed to few-shot or zero-shot Chain-of-Thought (CoT) prompting. |
| Approach: | They propose a decoding strategy that nudges LLMs to continue reasoning, thereby preventing immature reasoning processes. |
| Outcome: | The proposed method significantly improves LLM reasoning capabilities on diverse reasoning benchmarks. |
Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent prompting techniques have improved LLMs’ performance on various reasoning tasks, but there is little understanding of what triggers reasoning abilities in LLM in the inference stage. |
| Approach: | They propose a method that transforms a natural language problem into code and directly prompts the LLM using the generated code without resorting to external code execution. |
| Outcome: | The proposed method boosts multiple LLMs by 22.52 percentage points on GPT 3.5, 7.75 on Mixtral, and 16.78 on Mistral. |
Puzzle Solving using Reasoning of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their logical reasoning abilities across various domains. |
| Approach: | They propose to divide puzzles into rule-based and rule-less categories and critically assess LLMs' performance through various methodologies. |
| Outcome: | The proposed models have demonstrated capabilities in deductive reasoning and inductive reasoning, but they face limitations in inductive thinking. |
HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing prompting techniques for providing control are task-specific and lack generality; this limits their adoption among non-expert users. |
| Approach: | They propose a prompting strategy that encourages large language models to help non-expert users by asking relevant questions and leveraging user answers to execute a task. |
| Outcome: | The proposed prompting strategy is able to help non-expert users with a variety of tasks. |
From Reading to Compressing: Exploring the Multi-document Reader for Prompt Compression (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have recently exhibited performance gains owing to a wide variety of prompting techniques, including Retrieval-Augmented Generation (RAG), Chain-of-Thought (CoT), and In-Context Learning (ICL). |
| Approach: | They propose a prompt compression method that captures the global context without compromising semantic consistency while detouring the necessity of pseudo-labels for training the compressor. |
| Outcome: | Empirical results show that the proposed method retains key contexts while reducing the prompt length by 80%. |
BenNumEval: A Benchmark to Assess LLMs’ Numerical Reasoning Capabilities in Bengali (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in general-purpose tasks but struggle with numerical reasoning, especially in low-resource languages like Bengali. |
| Approach: | They propose a benchmark to assess LLMs on numerical reasoning tasks in Bengali. |
| Outcome: | The proposed benchmark assesses LLMs on numerical reasoning tasks in Bengali. |
From Observation to Understanding: Front-Door Adjustments with Uncertainty Calibration for Enhancing Egocentric Reasoning in LVLMs (2025.findings-acl)
Copied to clipboard
Shenshen Li, Wenxin Meng, Lei Wang, Hao Yang, Chong Peng, Peng Yan, Fumin Shen, Jingkuan Song, Heng Tao Shen, Xing Xu
| Challenge: | Existing methods that adapt LVLMs to egocentric tasks overlook critical agent-environment interactions, limiting their ability to perform egoic reasoning. |
| Approach: | They propose a zero-shot paradigm to enhance egocentric reasoning by simulating human causal reasoning by formalizing ego-centric reasoning using a structural causal model. |
| Outcome: | The proposed method improves egocentric reasoning abilities on six tasks. |
MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation (2025.acl-long)
Copied to clipboard
María Andrea Cruz Blandón, Jayasimha Talur, Bruno Charron, Dong Liu, Saab Mansour, Marcello Federico
| Challenge: | Existing benchmarks focus on English or use translated data, which fails to capture cultural nuances. |
| Approach: | They propose to use a multilingual end-to-end Meta-Evaluation RAG benchmark MEMERAG to assess accuracy and faithfulness of RAG systems. |
| Outcome: | The proposed benchmark can identify improvements offered by advanced prompting techniques and LLMs. |